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Abstract 

Hie results of a citation pattern analysis on the journal Educational Technology Research and 
Development (ETR&D) for the period 1990-99 are presented. Reference lists for the 260 articles published 
during that period were compiled along with over 1,600 citations of those articles in other social science 
journals. Breakdowns of most commonly cited references during that period are presented along with an 
analysis of the journals more cited by and most citing of ETR&D. 

Introduction 

I want to see the book you stole there after reading it, to keep others from reading it, and you hid it 
here, protecting it cleverly, and you did not destroy it because a man like you does not destroy a book, but 

simply guards it and makes sure no one touches it. I want to see , the book that everyone has 

believed lost or never written, and of which you hold perhaps the only copy. 

Brother William of Baskerville 
In Umberto Eco’s The Name of the Rose 

There is a growing mountain of research. But there is increased evidence that we are being 
bogged down today as specialization extends. The investigator is staggered by the findings 
and conclusions of thousands of other workers — conclusions which he cannot find time to 
grasp, much less reme mber, as they appear. . . A record, if it is to be useful to science, must be 
continuously extended, it must be stored, and above all it must be consulted. 

Vannevar Bush 

“As We May Think” in Atlantic Monthly, 1945 

Social science research is itself a social endeavor. Although media representations often depict 
researchers as solitary figures struggling independently to discover “truth,” productive research is conducted 
within the framework of a community. “Research is complete only when the results are shared with the 
scientific community.” (APA, 2001, p. 3). In addition to collaboration with other researchers and the need to 
build upon (and sometimes tear down) the work of predecessors, the highest goals of research are defined by 
peer review. This examination of works by a qualified audience is required to ensure that conference 
presentations and published articles represent the best that the field has to offer. 

The examination of the artifacts of this process inform us of not only the perceptions and viewpoints of 
the individual authors, but also of these “peers” who serve as science’s gatekeepers. A number of researchers 
have exa mined the content of academic journals in order to capture trends or patterns of research behavior. 

Klein (1997) exa mined a nine-year period of publication of the development section of Educational Technology 
Research and Development (ETR&D). His content analysis indicated the “ID for Computer Technologies” and 
“Instructional Design & Development” were the most popular article topics and that almost half of all articles 
were descriptions of different activities with little or no supporting data. In a separate analysis, Klein (2002) 
performed a similar analysis on four years of research published in Performance Improvement Quarterly. In that 
analysis, about one-third of the articles contained empirical research. This follow-up research not only 
demonstrates that similar methodology can be applied to different journals in the field, but also that different 
journals in the same field may have uniquely differing publication patterns. In terms of looking for similarities, 
ETR&D was one of four journals examined by a group of researchers from the University of Kentucky (Anglin, 
Cain, Whitehouse, Cunningham, Newcomer, and Cunningham, 2003). The other three journals were the British 
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Journal of Educational Technology, Educational Technology, and the Journal of Research on Technology in 
Education. During the five year period examined (1997-2001), they identified the ERIC descriptors “the 
educational process: classroom perspectives” and “the educational process: societal perspectives” as the most 
used in indexing article content. Driscoll & Lee (2003) identified trends in distance education by analyzing 
article content for four selected journals over a five-year period. Among other results, they found that 
developmental research was rare and that case studies were the most prominent type of article. 

Citation analysis in particular provides an opportunity for authors to understand the influence and 
influencers of their work. At the level of an individual article, an author cites others so that the reader can see 
the social construction of knowledge. Publication represents successfully completing the peer review process, 
but citation of one’s work is a greater indicator of the influence of the work. Although those who cite may agree 
or disagree with an author’s work, citation denotes that the previous work was worthy of discussion. Citation 
analysis is less common than other types of research on publication patterns, but should become more 
accessible with improvements in on-line resources and databases. For example, Frisby (1998) examined citation 
patterns in five years of seven major journals in school psychology. An interesting aspect of this research was 
his definition of self-citation within a journal as “within -journal inbreeding,” a negative characteristic from his 
point of view. Another interesting example of citation analysis, was conducted by Oppenheim and Smith 
(2001). They focused on the citation patterns of final year students in an information science department. In 
direct contrast to “refereed” publications, their analysis indicated an increasing trend of students citing Internet 
resources rather than more traditional sources. Creamer (1998) examined citation patterns as a possible measure 
of faculty publication productivity. She reported that authors tend to cite more heavily other authors of the same 
gender. However, she suggested that this may be reflective of homogeneous gender informal professional 
networks rather than conscious bias. 



Limitations 

The current study is decidedly descriptive in nature. Also, the researchers believe that citations are a 
measure of importance, not the measure of importance. It must be added that not all citations are equal, their 
relative importance varies and a citation may be made to show agreement or disagreement with a particular 
point. However, it is suggested that patterns of citation data are informative, certainly as much as course surveys 
or opinion polls. 



Method 

For the current study, the journal Educational Technology Research and Development (ETRD) was 
examined over a ten-year publication period, 1990-99. This journal was selected because of its reputation in the 
field. For example, Maushak, Price, and Wang (2000) conducted a survey of 85 faculty members in the field of 
educational technology. According to their analysis, ETR&D was the overwhelming choice as the top journal in 
the field. During this period, 40 issues were published containing 260 journal articles. For the purpose of this 
study, only articles that were part of the Research or Development sections were considered. 

The tables of content for the issues were photocopied and used to uniquely identify each article and 
record relevant data into an Excel worksheet. One research , using the online version of the Social Science 
Citation Index (SSCI), performed a search on each article and recorded information on each citing article into a 
second worksheet (N=l,553). 

Another researcher created a photocopied record of the reference lists for the 260 articles. Due to the 
large number of references (N = 10,055), a number of individuals manually entered this reference data into a 
third worksheet. After these references were coded, one individual reviewed each entry in comparison with the 
printed record to ensure consistency and correct minor errors. 

All analysis was conducted manually and/or using the functionality of Excel (for example, to count or 
aggregate results). During analysis, the photocopied indices and reference lists were consulted to clarify and/or 
correct entries. 

As with any intensive data entry endeavor, errors were made in collecting and recording the data. All 
attempts were made to correct identified errors and the impact on aggregrated results should be minimal. 

The limited journal scope (only ETR&D) and publication period (10 years) were selected arbitrarily by 
the researchers. Due to the descriptive nature of the analysis, any inference or interpretation must be made 
cautiously. 
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Results 

Of the 260 core journal articles, 94 (36.15%) were authored by one individual. Two authors were 
involved in the writing of 98 articles (37.69%). Sixty-eight articles (26.15%) were written by three or more 
authors with nine authors being the most for any article. 

The researchers attempted to establish the gender of the various authors. This was largely 
idiosyncratic. A large number of authors were identified via professional contact or a gender-specific name. 
Others were identified by locating biographical information on the Internet. For the 94 articles written by one 
author, 58% were male; 33% were female; and 9% were unknown. ETR&D uses a blind review process in 
which the identity and gender of the author are unknown. Without additional knowledge of submissions by 
gender or general gender makeup of the field, it is impossible to determine specific causes for the lower 
publication numbers for females. For the 166 articles of multiple authorship, 44% included at least one male 
and one female; 27% had exclusively male authors; 9% had exclusively female authors; and 20% could not be 
determined. The large number of undetermined makes analysis difficult. The large number of mixed gender 
efforts is encouraging. The large discrepancy between male -only (27%) and female -only (9%) efforts may again 
reflect gender distribution in the field or preferences in informal networking. 

Author productivity was classified according to order of authorship (see Table 1). Those that had the 
largest number of first author articles were arbitrarily labeled “The Prolific” to represent the primary importance 
given to first authors in academic communities. Those with the largest number of second author articles were 
labeled “The Mentors,” suggesting their role in helping others with attaining first author credits in this journal. 
Lastly, those that had the greatest number of authorships of third or later priority were labeled “The 
Collaborators,” tenuously identifying them for a more minor yet continuing role in assisting in authorship. 

Table 1. Author Productivity 

The Prolific (First Authors) Number 

Mable B. Kinzie 
Lloyd Rieber 
Martin Tessmer 

The Mentors (Second Authors) Number 

Michael J. Hannafin 
Howard J. Sullivan 
James D. Klein 

The Collaborators (Third... Authors) Number 

Gary R. Morrison 
John F. Wedman 
John Bransford 



of Articles 

6 

6 

6 

of Articles 

8 

7 

5 

of Articles 

4 

4 

3 



Of the 10,055 references cited by the core articles, 53% were to journal articles; 36% were to books; 
and 11% were classified as other (e.g. websites, technical reports, dissertations, etc.). The most cited journals 
and most cited journal article authors are presented in Table 2. 



Table 2. Most Cited Journals and Journal Article Authors 



Journals 

ETR&D 

Educational Technology 
Journal of Educational Psychology 
Review of Educational Research 
Educational Researcher 
Journal Article Authors 
David H. Jonassen 
Michael J. Hannafin 
Gavriel Salomon 
Robert D. Tennyson 
Richard E. Mayer 
Steve M. Ross 



Number of References 

494 (9.40%) 

351 ( 6 . 68 %) 

331 (6.30%) 

177 (3.37%) 

163 (3.10%) 
Number of References 
76(1.45%) 
64(1.22%) 
60(1.14%) 
59(1.12%) 
58(1.10%) 
58(1.10%) 
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The references to books were further subdivided into edited and non-edited books. The most cited for 
both categories are shown in Table 3. For this analysis, multiple editions of a work were counted together (the 
year of the earliest edition is provided in the table). 



Table 3 . Most Cited Edited and Non -edited Books 








Number of 


Edited Books 


Editor 


References 


Instructional Design Theories and 
Models: An Overview of Their 


C.M. Reigeluth 


55 


Current Status (1983) 
Instructional Technology: 


R.M. Gagne 


34 


Foundations (1987) 

Handbook of Research on Educational 


D.H. Jonassen 


33 


Communications and Technology 
(1996) 

Instructional Technology: Past, 
Present, and Future (1991) 


G.J. Anglin 


31 


Knowing, Learning, and Instruction: 
Essays in Honor of Robert Glaser 


L.B. Resnick 


28 


(1989) 




Number of 


Non-edited Books 


Authors 


References 


The Systematic Design of Instruction 


W. Dick 


40 


(1978) 


L. Carey 




The Conditions of Learning and 
Theory of Instruction 


R.M. Gagne 


39 


Principles of Instructional Design 


R.M. Gagne 


35 


(1974) 

Cooperative Learning: Theory, 


R.E. Slavin 


14 


Research, and Practice (1983) 

Mind in Society: The Development of 
Higher Psychological Processes 


L.S. Vygotsky 


13 


(1978) 

Teaching for Competence (1983) 


H. Sullivan 


12 



The 260 core articles were entered into the Social Science Citation Index (SSCI) in late 2003 to 
determine in which journals these articles were cited. Table 4 displays the most citing journals and the core 
articles that were the most cited. Because the core articles were published over a period of 10 years and the 
SSCI data represented a snapshot of all citations to that point, it was determined that average citations since 
year of publication would provide the most unbiased measure of influence. 



Table 4. Most Citing Journals and Most Cited Articles 



Number of 

Citing Journals Citations 

ETR&D 475 (30.59%) 

Journal of Educational Computing Research 100 (6.44%) 

Computers in Human Behavior 59 (3.80%) 

Instructional Science 51(3.28%) 

British Journal of Educational Technology 50 (3.22%) 

Most Cited Articles Citations/Yr 

Media Will Never Influence Learning by R.E. Clark 7.78 

Objectivism versus Constructivism: Do We Need a New 4.83 

Philosophical Paradigm by D.H. Jonassen 

Instructional Design for Situated Learning by M.F. Young 4.40 
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Will Media Influence Learning? Reframing the Debate by 4.22 

R.B. Kozma 



Conclusion 

These preliminary results provide guidance for professionals and students in the field as to what some 
consider the influential works and individuals with which one should be familiar. The methodology used 
advances what has previously been published in the form of course/program surveys and opinion polls. 

A more complete presentation is currently being developed for eventual publication. The research team 
has also begun to collect data on related journals during the same time period. This will allow comparison and 
triangulation with this dataset. Also, data collection has begun with ETR&D for the next five-year period (2000- 
2004). This will allow chronological comparison with the present dataset. Finally, the improvement of online 
tools continues to make the process more manageable. As journals and databases add functionality and online 
content, more advanced analysis will become increasingly automated and informative. 
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